Papers with LSTM-based models
You Only Need Attention to Traverse Trees (P19-1)
Copied to clipboard
| Challenge: | Recent research has focused on sentence representations. |
| Approach: | They propose a tree-based model that captures phrase-level syntax and word-level dependencies by doing recursive traversal with attention. |
| Outcome: | a new model captures phrase-level syntax and word-level dependencies with attention. |
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate layer matching are limited due to huge over-parameterization . |
| Approach: | They propose to match intermediate layers of teacher and student in output space via attention-based layer projection. |
| Outcome: | The proposed method outperforms existing methods on GLUE tasks. |